Measuring Application Domain Knowledge: Results from a Preliminary Experiment

Conceptual models are used in IS development for capturing and specifying requirements. However, the mere understanding of the syntax or semantics of a modeling language is not the most crucial factor. More relevant is pragmatic knowledge about the application domain. The problem that this paper addresses is how one can verify that a shared understanding of the application domain exists. In our study we show that domain-specific languages are an indicator for separating novices from experts in a given application domain. Novices and experts can be distinguished based on the domain-specific language they use. We demonstrate that these different language communities can be observed empirically by employing latent semantic analysis (LSA) as an instrument and by measuring semantic similarity. The separation of groups using LSA is also possible if the terminology, the application domain, or the expert-layperson-status of the examined group are unknown. Therefore the separation based on domain-specific languages is independent of the domain under consideration or the prior knowledge of the researcher. This provides a useful measurement instrument for studying the role of application domain knowledge in future research.